Three-dimensional nematic spin liquid in the stacked triangular lattice 6H-B structure 
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Recently, a number of experiments indicate the possible presence of spin liquid phases in quantum 
magnets with spin-1/2 and spin-1 moments sitting on triangular-lattice-based structures. In relation 
to recent experiments on Ba3CuSb20g and Ba3NiSb209, several theoretical proposals have been 
made for spin liquid phases and spin-liquid-like behaviours on the stacked triangular lattice. While 
the crystal structures of these materials are currently under debate, it is nonetheless interesting to 
understand possible spin liquid phases on such frustrated lattices. In this work, we apply Schwinger 
boson mean-field theory and projective symmetry group (PSG) analysis to investigate spin liquid 
phases on the fully three-dimensional 6II-B structure, in contrast to previous works that considered 
two-dimensional systems. We find that a nematic Z2 spin liquid phase, where the lattice-rotational 
symmetry is spontaneously broken, is the most promising spin liquid phase that is consistent with 
spiral magnetic ordering in the classical limit. We discuss the implications of our results to future 
theoretical and experimental works. 

PACS numbers: 75.10.Jm, 75.10.Kt 



I. INTRODUCTION 

Spin-1/2 moments on two or three dimensional lat- 
tices may not order, even at zero temperature, owing 
to competing interactions and/or quantum fluctuations 
and form a highly entangled state of quantum matter. ^"'^ 
Such a correlated state, called a quantum spin liquid 
(QSL),^"^ is capable of supporting quasiparticle excita- 
tions that carry only a fraction of the quantum numbers 
of the underlying electrons that make up the spin system. 

This possibility of emergent quantum number frac- 
tionalization and other exotic excitations like artifi- 
cial photons,'^ resulting from strong many-body corre- 
lation effects in condensed matter systems, have fu- 
elled massive interest in quantum spin liquids both 
theoretically and experimentally. A large number of 
Mott insulators in various frustrated lattice geome- 
tries have been investigated, and several candidate ma- 
terials have been identified. Most of these materi- 
als, such as K-(BEDT-TTF)2Cu2(CN)3,^ Cs2CuCl4,i", 
EtMe3Sb[Pd(dmit)2]2", (spin-1/2 on triangular lat- 
tice) and Herbcrtsmithite (ZnCu3(OH)6Cl2, spin-1/2 
kagome lattice)^^'^'^ are two dimensional. However, some 
three dimensional systems, like the hyper-kagome lat- 
tice Na4lr308,^'' several candidate quantum spin-ice ma- 
terials on the pyrochlorc lattices, and more recently 
the stacked compounds Ba3CuSb209^*^'^^ (S=l/2) and 
Ba3NiSb20g^^ have shown much promise. In light 

of the above developments, it is useful to explore and 
understand the behaviour of quantum spin liquids in as- 
sociated frustrated three dimensional lattices. 

In this paper, we study bosonic spin liquids that can 
be realized in quantum magnets on a three dimensional 
crystal structure built by stacking triangular lattices in a 
staggered fashion. This so-called 6H-B phase is obtained 
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FIG. 1. The 6II-B lattice structure and coordinate system 
used throughout the paper. The 6H-B lattice consists of the 
AB-stacked triangular lattices in three dimensions. The left 
figure shows neighboring two sublattices projected into the 
afe-plane. For the B sublattice, the couplings with Ji are 
omitted to simplify the figure. The right figure shows the 
lattice structure along the c-axis. The ellipse indicates the 
unit cell located at the origin of the coordinate system. 



in the P63 /mmc lattice structure, as shown in Fig. 1. 
In particular, we focus on a spin-1/2 antiferromagnetic 
Heisenberg model with intra-layer and inter-layer spin 
exchanges. The two interactions compete and the model 
is frustrated when the intra and inter-layer exchanges 
are comparable. While earlier studies on the 2D honey- 
comb limit found interesting ground state degeneracies 
and non-collincar magnetic states selected by quantum 
order-by disorder,^"* our Schwinger boson mean field the- 
ory (SBMFT) captures a 3D Z2 spin liquid, a 3D U{1) 
spin liquid and a layered, effectively 2D, Z2 spin liquid in 
addition to the usual magnetically ordered phases. Us- 
ing projective symmetry group (PSG) classification for 
the spin- liquids^ , we find the above bosonic spin liquids 
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that arc consistent with various lattice symmetries. A 
central finding of our SBMFT study is that a 3D nematic 
Z2 spin liquid, which spontaneously breaks lattice rota- 
tion symmetries, is found to be energetically more stable 
than an isotropic Z2 spin liquid in the corresponding pa- 
rameter regime. Interestingly, we find that, unlike the 
isotropic case, the nematic spin liquid is naturally con- 
nected to the classical magnetic orders. The breaking of 
lattice rotation symmetry in the nematic spin liquid bears 
a characteristic signature in the two-spinon excitation 
spectrum for the nematic spin liquid. Such two-spinon 
spectrum is relevant to neutron scattering experiments. 
Within mean-field theory we find several quantum phase 
transitions among spin-liquids, and between spin-liquids 
and magnetically ordered phases. While some of them, 
like the transition between the collincar Nccl and the 3D 
U{1) spin liquid, arc most likely rendered discontinuous 
by gauge fluctuations beyond the mean field, others like 
the transition between the 3D, Z2 and the spiral are likely 
to remain continuous. The transition between various 
magnetically ordered phases and the spin liquids can be 
understood in terms of condensation of the spinons^"'''^^ 
and a charge-2 Higgs field (discussed later). 

As noted earlier, a partial motivation for study- 
ing the above spin model in this lattice structure 
stems from recent interests in two new candidate QSLs, 
viz., BaaCuSbaOg (spin-1/2) and (6H-B-)Ba3NiSb209, 
(spin-1) where initial experiments suggested a 6H-B 
structure. These compounds do not show magnetic 
order down to a few hundred niK, despite Curie- Weiss 
temperatures, as measured from the high temperature 
susceptibility, greater than 50 K. Furthermore, they show 
a large intermediate temperature window where the mag- 
netic specific heat shows a linear temperature depen- 
dence suggesting the possibility of an unconventional 
spin state. While recent experiments on the spin-1/2 
Cu compound revealed a different crystal structure for 
Ba3CuSb209,^^'^^'-^'^ it may be still useful to think about 
possible spin-1/2 and/or spin-1 spin liquids on the 6H-B 
phase in search of three dimensional spin liquids and pos- 
sible exotic quantum phase transitions. This approach 
is in contrast to the previous theoretical studies that 
mainly considered two dimensional spin models for the 
compounds. ^^^^'^ 

The rest of this paper is structured as follows. In Sec. 
II, we give an overview of the PGa/mmc lattice struc- 
ture obtained in the 6H-B phase, discuss its symmetries, 
and introduce a frustrated Heisenberg model. Next, we 
discuss the Schwinger boson formalism in Sec. Ill, and 
derive the mean-field Hamiltonian, which can describe 
the transition between classical order and a regime of 
stronger quantum fluctuations. To classify distinct mean- 
field ansatze with a given symmetry, we introduce the 
PSG analysis in Sec. IV, and describe the allowed ansatz 
with the full symmetry of the lattice, along with those 
that break lattice rotational symmetry. We work out 
the magnetic phases obtained in the classical limit of our 
model in Sec. V, and note that this behaviour cannot be 



fully captured from the symmetric spin liquid ansatz. In 
fact, we find that the symmetric spin liquid is generally 
energetically inferior to the nematic spin liquid in the rel- 
evant part of the phase diagram. The full phase diagrams 
as a function of k and J2/ Ji for the different ansatze are 
discussed in Sec. VI. Wc also point out the nature of 
different phase transitions and indicate the underlying 
mechanisms to study them. Finally, the structure of the 
two-spinon excitations is discussed in Sec. VII. We end 
by summarizing our results and discussing their implica- 
tions in Sec. VIII. The details of the PSG calculations 
and derivations of the ansatze are left in different Appen- 
dices. 



II. LATTICE STRUCTURE AND SPIN 
HAMILTONIAN FOR THE 6H-B PHASE 

The 6H-B structure may be thought of as a collection 
of triangular lattices stacked along the c-axis, with two 
consecutive layers being offset as shown in Fig. 1. Sites 
on even layers form the triangular A sublattice. Those on 
odd layers form the triangular B sublattice, and are offset 
above the centre of the triangles of the A sublattice. 

On such a lattice, we consider a spin-1/2 model where 
the predominant interactions between spins on the lattice 
are nearest-neighbour antiferromagnetic super-exchange 
interactions. Interactions within the A and B triangular 
planes have strength Ji , and those between neighbouring 
planes have strength J2. Both couplings may be compa- 
rable in the case of systems such as Ba3NiSb20g; where 
the exchange paths are mediated through intermediate 
oxygen atoms. The resulting Heisenberg Hamiltonian is 

i? = Ji ^ S,-S, + J2 

(ij) in plane j) between planes 

where Ji, J2 > 0. 

This model has two simple limits: (i) Ji 3> J2 and (ii) 
Ji <C J2- In the first case, the model is reduced to the 
two-dimensional triangular lattice Heisenberg model. It 
is well known that the ground state of the model has 120° 
non-coUincar (spiral) Neel order. In the other limit, the 
system has a three-dimensional bipartite lattice structure 
without no frustration so that it allows collincar Neel or- 
der as its ground state. However, in the regime where 
both couplings are comparable (Ji ~ J2), the interac- 
tions compete with each other, and this may lead to a 
quantum spin liquid ground state. At this point, we also 
note (as discussed in detail later) that in the classical 
limit, the above spin model has the spiral ordered ground 
state in the intermediate region of coupling constants (see 
Fig. 3). 

We proceed by noting various symmetries of the spin 
model in anticipation of our future Projective Symme- 
try Group calculation. The lattice is described by the 
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following primitive vectors, as seen in Fig. 1: 

Ri ax, R2 = a ^ "^^^ ' ^'^^ 

where a and c are the sublattice spacings in the a-h plane 
along the c-axis, respectively. Wc write lattice coordi- 
nates as va = J2n=i fri'Rn = , ?'2 , ''s ) A for the A sub- 
lattice, and tb = +rBA= (''i, ''2, '"3)3 for the B sub- 
lattice, where tba ~ — Ri/3 — 2R2/3 + R3/2. In this 
coordinate representation, ?■„ (n ~ 1,2,3) is an integer. 
The system has the following symmetries: spin-rotation, 
time-reversal, and space group symmetries. The first two 
symmetries are discussed in the next sections. The spin 
model (1) has the space group Pds/mmc with following 
seven generators 

• Translations Ti and T2 within the triangular plane, 
and another, T3, along the c-axis. 

• A 120°-rotation R around the z-axis, centered on 
an A site. 

• A reflection Hi through the y-z plane, and another, 
112, through the x-y plane. 

• An inversion S through the midpoint between 
neighbouring A and B sites. 

Consequently, the space group transformations are de- 
fined as 



T2 
T3 
R 

Til 
TI2 



{ri,r2,r3)p 
{ri,r2,r3)p 

{ri,r2,r3)p 
{ri,r2,r3)p -> 
{ri,r2,r3)p 



ri + I,r2,r3)p, (3a) 
ri,r2 + I,r3)p, (3b) 
ri,r2,r3 + l)p, (3c) 
-?'2 + Sp^B,ri -r2 + Sp^B, r3)p, 

(3d) 



-j'l +r2,r2,T'3)p, 
ri,r2, -rs - (5p,B)p, 
-»'2, -r3)p, 



(3c) 
(3f) 
(3g) 



where p (p) ~ A, B {B, A) refers to the two sublattices. 

Having determined the symmetries of our spin model, 
in the next section we begin our Schwinger boson mean- 
field analysis, and proceed to identify the various possible 
bosonic spin liquids within a projective representation of 
the above symmetry group. 



III. SCHWINGER BOSON MEAN-FIELD 
THEORY AND GAUGE STRUCTURE 

In the Schwinger boson mean-field theory, the 
spin operator at site i is represented in terms of bosonic 
spinous bifj_: 



(4) 



where a = x,y,z, fi,iy =t,i, {ct"} are the Pauh matri- 
ces, and a sum over repeated Greek indices is assumed 
hereafter. In this formalism, operators of any spin quan- 
tum number S can be represented, which is given by the 
on-site density constraint. 



2S. 



(5) 



Within the mean-field theory, the constraint is imple- 
mented on average. In general, k/2 represents the spin 
quantum number and acts as a parameter that deter- 
mines the degree to which quantum fluctuations arc im- 
portant to our Hamiltonian. The k (~ S*) — > 00 limit 
corresponds to classical limit. "'^ 

Following the usual SBMFT techniques, "^^'''^ we now 
consider the mean-field decoupling of the Heisenberg 
terms. With the help of the constraint (5), we can rewrite 
these terms as 
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(6) 



where fjij = bif^e^j^^bju and Cp^u is totally antisymmetric 
tensor with e-\i = 1. In this form, the mean- field de- 
coupling is straightforward. We introduce the parame- 
ter Xi^ = {bi^l) which represents the bosonic condensate 
fraction. If 7^ 0, there is condensation of the spinous, 
and hence a long range magnetically ordered state with 
spontaneously broken spin-rotation symmetry. However, 
if the spinous arc gapped then there is no condensate, 
and the spin-rotation symmetry is preserved. This is a 
spin liquid state with gapped bosonic spin-1/2 (spinon) 
excitations. In three dimensions, it can be either a Z2 or 
U(l) spin liquid. In the former case, in addition to the 
spinous, there is an emergent gapped non-magnetic ex- 



citation called the 



On the other hand, in the 



U(l) spin liquid, there is an emergent gapless photon 
(with two polarization modes) and a gapped magnetic 
monopole excitation.^ These issues are discussed in some 
detail later. A Lagrange multiplier, A^, is used to imple- 
ment the constraint (5). The mean-field Hamiltonian is 
written as 



Hmf = f]\j + H-c. 

i>3 ^ ^ 

i 

i>j \ ^ 



i>j 



(7) 



where rjij = {fjij) is the mean- field expectation value of 
the bond parameter. The above mean-field Hamiltonian 
is then solved self-consistently using the following saddle- 
point equations: 



d{H_ 



MF 



0, 



MF 



dx* 



0, 



d {Hmf) 



0. 



(8) 
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There exists a !7(1) gauge redundancy in the Schwinger 
boson representation of the spin operator (4). That is, 
under the transformation 

^ e">-&,^, (9) 

the spin operator Si in (4) is invariant. However, the 
mean-field parameter transforms as 

->e-*(*'+'^^)?7,y. (10) 

For non-bipartite lattices, when a particular mean-field 
state is chosen such that 77,^ 7^ is fixed, the above 
U{1) gauge invariance is broken down to Z2, since now 
the mean-field Hamiltonian is gauge invariant only for 
(j)i = 0, TT.^''^"' The resulting state is a Z2 spin liquid. 
The gauge redundancy described above, however, sug- 
gests that different Z2 spin liquid ansatzc may be con- 
nected by gauge transformations and hence correspond 
to the same physical state. 

In the next section, we provide the classification 
scheme for physically distinct Z2 spin liquid phases, using 
a PSG analysis.2>34 

IV. PROJECTIVE SYMMETRY GROUP 
ANALYSIS FOR THE MEAN-FIELD ANSATZE 

A. Brief Overview 

At the mean-field level, different spin liquids are char- 
acterized by different PSGs, which arc projective exten- 
sions of the symmetry group of the Hamiltonian. ^■'^^'^'^ 

For a given spin liquid ansatz, (in our case a choice 
of r]ij) the PSG is the set of operations {GxX}, where 
X is an element of the symmetry group and Gx is the 
associated gauge transformation, that leave the mean- 
field Hamiltonian invariant. Here, we will describe the 
general framework to determine {Gx} for a particular 
symmetry group (SG). The details of the calculations 
are given in Appendix A and B. Note that this analysis 
characterizes spin-liquid states without long-range order; 
that is, mean-field Hamiltonians (7) in the absence of a 
condensate Xi^. Considering the case of Z2 spin liquids, 
we define 

% = mjbl,<^t.ub]^, (11) 

and find that under the combined effect of a symmetry 
transformation, X, and associated gauge transformation 
Gx, we have 

[Ux^ jiitj — r]ije °x(i)tj.^t^^''x(j)v- 

(12) 

Hence, for {GxX} that leave Hmf invariant, symmetry- 
related mean-field parameters are found by 

r7xwxo-)=^,.e-'(^-[^«l+^-l^«». (13) 



As mentioned in Sec. HI, gauge transformations gen- 
erally change Hmf, but a particular subset, called the 
Invariant Gauge Group (IGG) of the PSG, may leave it 
invariant. The IGG is a projective extension of the iden- 
tity operation of the symmetry group. One can choose a 
gauge in which the IGG is independent of the site.^ For 
Z2 spin liquids, IGG={-|-1, — 1}. The structure of the 
IGG determines the nature of the low-energy gauge fiuc- 
tuations around Hmf- For Z2 spin liquids, these fluc- 
tuations are gapped. There is a gapped non-magnetic 
vortex-like excitation that carries the fiux of the Z2 gauge 
field. However, for U{1) spin liquids, there is an emergent 
gapless photon, as well as a gapped magnetic monopole 
excitation, the latter resulting from the compactness of 
the U(l) gauge group. ^ 

We note that if GxX leaves Hmf invariant, so does 
WGxX for W e IGG, which means that a class of Gx is 
defined only up to elements of the IGG. To generate re- 
strictions for {Gx}, we note that products of Gx^^a 
that are physically equivalent to an identity transfor- 
mation must be in the IGG.^'^'' Since each operation 
leaves Hmp invariant, the total transformation has no 
net physical transformation and also leaves Hmf in- 
variant. Multiplication rules among the generators of 
the symmetry group generate precisely these products 
of transformations. We write these rules in the form 
XaXi,X~^X'^'^ = /, and note that upon inclusion of the 
corresponding gauge transformations, we have 

{GxM[Gx,Xb){Gx^X,)-\Gx,Xd)-^ ^IGG. (14) 

Motivated by the classical solution of the Heiscnberg 
model (1) (as determined in Sec. IV), we consider two 
sets of symmetry groups. 

SGi = Span{Ti,T2,r3,Hi,H2,S}, (15a) 
SG2 = Span{Ti,T2,r3,Hi,H2,S,i?}. (15b) 

The group SG2 consists of the full set of lattice symme- 
tries given in (3). However, we will find that the resulting 
ansatz with SG2 is too restrictive to be consistent with 
the classical limit of this model. We find that it does 
not lead to the right magnetic ordered state in the clas- 
sical limit. Thus, we also consider the symmetry group 
SGi(c SG2) by removing the rotation i?, which will be 
shown to provide a spin liquid phase consistent with the 
spiral magnetic order in the classical limit. For each of 
these symmetry groups, we find the gauge transforma- 
tions {Gx} associated with the symmetry operations, 
and generate the resulting ansatze Hmf using Eq. (13). 



B. Z2 Spin Liquid Ansatze 

In this subsection, we discuss the ansatze for both 
symmetry groups SGi and SG2. A brief sketch of their 
derivation is given in Appendix C. Mean-field Hamilto- 
nians for the ansatze arc provided in Appendix D. 
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0-flux ansatz Ti-flux ansatz 




rjjp r|,„ r|,p 
— > — — ^ — — ^ — — * — 

FIG. 2. Mean-field ansatze of the symmetry group SGi, show- 
ing the directions and magnitudes of the allowed mean-field 
parameters, viewed along the c-axis. Filled and empty circles 
denote A and B sublattices, respectively. Arrows indicate the 
directions of the four allowed positive, real mean-field param- 
eters r]ia, Tjifi, i]2a, aud 1^213 ■ Tlic sliaded region indicates the 
inside of the three-dimensional path around which the gauge 
fiux in (16) is defined. 

a. SGi: For the symmetry group SGi, we find two 
ansatze, named the 0-flux ansatz and tt-Aux ansatz. The 
ansatze are depicted in Fig. 2. Both ansatze are trans- 
lationally invariant. To explicitly preserve time-reversal 
symmetry, we work in a gauge where these ansatze are 
real- valued mean-field parameters rjij. Since rjji = —rjij, 
we denote the directions of the positive parameters in 
Fig. 2. In the symmetry group SGi, due to the lack 
of rotational symmetry, there are two different in-plane 
mean-field parameter magnitudes rjia and 771^, and two 
inter-plane ones r]2a and r]2^. 

The ansatze are differentiated by the flux of rjij 
through the inter-plane loop shown in Fig. 2. This gauge 
flux <& on the closed loop is defined through 

Viit2{~Vi2iJ^1^3i4i~VU,) = ??la'7l/3'/2a'72/3e'*. (16) 

Under the effects of a gauge transformation, (10), this 
flux is invariant, providing a clear way to distinguish be- 
tween these two ansatze that have the same symmetry 
group. '^^ The name of each ansatz comes from the above 
gauge-invariant flux $ = or tt. 

The gauge flux for the loop in Fig. 2 is defined only 
when every r]ij along the loop is non-zero. In the mean- 
field theory, we will come across two special cases with 
Via = Vii3 = m and = V213 = V2- one where tji = 0, 
and the other where 772 = 0. In the first case, rji = 
0, our 0- and 7r-flux ansatze become gauge-equivalent. 
Furthermore, the connectivity changes so that the two 
sublattices become bipartite. As we show later, in this 
case we actually have a three-dimensional gapped U{1) 
spin liquid, which is in principle a stable phase, unlike 
in two dimensions. We denote this as the 3D-[/(l) state. 
In the second case, 772 = Oj our two ansatze are identical, 
giving a two-dimensional Z2 spin liquid with zero fiux 
through rhombus plaquettes in the triangular planes. We 



denote this as the 2D-Z2 state. It is one of the two Z2 
spin liquids allowed in the symmetry of the 2D triangular 
lattice. The symmetry group of the 6H-B structure does 
not include the 2D triangular lattice symmetry group 
as a subgroup, due to the lack of the six-fold rotational 
symmetry. As a result, the PSG analysis on the 611- 
B structure, in the 2D limit, recovers only the Z2 spin 
liquid with zero fiux within the rhombus plaquettes in 
the triangular planes. 

b. SG2: The PSG analysis of the full symmetry 
group SG2 finds only one allowed ansatz, which has tt 
fiux. This is related to the 7r-flux ansatz with SGi, for 
the case where the mean-field magnitudes acquire rota- 
tional symmetry, i.e., r/ia = 771/} and r]2a = V2p- Thus, 
the TT-flux ansatz for SGi includes the ansatz for SG2 as 
a special case. 

Having discussed the allowed ansatze for our symmetry 
groups SGi and SG2, we consider the phases obtained in 
different parameter regimes. 



V. CLASSICAL ORDERING 

We begin by studying the classical ground state of the 
spin model (1), in the context of Schwinger boson mean- 
field theory. The classical limit is obtained by taking the 
K — >■ 00 limit of the SBMFT. From the scaling behavior 
of the mean-field solution for k ^ 1, (77^- ~ k, ^ ^/k, 
X ^ k) the ground state energy can be written as"^^ 




+ Y.~Xr{\£.,\'-l), (17) 

i 

In the above expressions, fjij = rjij/n, Xi^ = Xi^/y/n, 
A = A/k. The ground state is determined by solving the 
following mean-field equations: 

I^^O, ^=0. (18) 

Making use of the above mean-field equations, we rewrite 
the ground state energy as 

^^ = ExS^s^=+^A,(|sJl-l), (19) 

i>j i 

where the classical spin vector is given by 

= x*^(Tf^yXiv. (20) 

This is precisely the classical Heisenberg model with the 
constraint of normalized spin vectors. The solution of 
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FIG. 3. Phase diagram of classical magnetic order of the 
Hamiltonian (1) as a function of J2/J1. 
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plane has 120° Neel order with Qc=±K=±(^, 0, 0) in 
the first Brillouin zone. Conversely, when J2/J1 > 3, 
Qc = 0, and there is intcr-plane coUinear Neel order. 
In the intermediate case, < J2/J1 < 3, the degenerate 
spiral ordering wavevectors interpolate between these be- 
haviors, and are shown in Fig. 4. 

Throughout our classical Luttinger-Tisza calculations, 
we find that Qc • i = 0. Hence, for the classical order, we 
can consider the system as comprised of two layers, which 
has a honeycomb lattice structure when looked at along 
the c-axis. Thus, our classical solution is equivalent with 
those of the J1-J2 Heisenberg model on the honeycomb 
lattice.21'24 

The classical spiral states for < J2I J\ < 3 sponta- 
neously break lattice rotational symmetry R, as well as 
spin-rotational symmetry. In the next section. Sec. VI, 
we will see that the 7r-flux ansatz with the full symmetry 
of the lattice does not host a condensate that gives any 
such spiral order in the classical limit. In contrast, the 
0-flux ansatz does so, and selects the particular states 
shown in Fig. 4 (blue line). 

With the classical limit understood, we now present 
the phase diagram through the parameter range of quan- 
tum fluctuations k. 
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VI. MEAN-FIELD PHASE DIAGRAM 



FIG. 4. (Color online) Magnetic ordering wave vectors (Qc) 
of the classical solutions. The wave vectors for the degenerate 
states at a given J2 / Ji are plotted with a red line in the mo- 
mentum space, and are determined by (22). The degeneracy 
is lifted by the quantum order by disorder effect. We show 
the selected states within the 0-flux ansatz with a blue line, 
which are described in Sec. VI. The black hexagon denotes 
the first Brillouin zone at fc^ = 0. 



In this section, we study the phase diagram for the 
different ansatze of symmetry groups SGi and SG2, as a 
function of k and J2/J1, where k — ^ 00 is the classical 
limit. We will begin by considering the 0-flux and tt-Aux 
ansatze of SGi. In the section following this, we will 
discuss the structure of the two-spinon minumum, which 
is relevant to neutron scattering experiments. 



(19) can be obtained via the Luttinger-Tisza method'*^ 
as follows: 

S'xir) ~ niCos[Qc • r] + n2sin[Qc • r], (21a) 
S%{r) = -niCos[Qc • r - e(Qc)] - n2sin[Qe • r - e(Qc)], 

(21b) 

where the unit vectors rii and n2 satisfy ni • n2 = 0. The 
ordering wave vector, Qc, and the relative phase, 0(Qc), 
are defined by the following equations: 

3 + 2cos[Qc • Ri] + 2cos[Qc • R2] + 2cos[Qc • (Ri + R2)] 
= (J2/J1)', (22a) 
(J2/Ji)e*®('^'=' = 1 + e''^--^^ + e*Qe-(Ri+R.2)_ (22b) 

The equations (22) are for the cases of J2/J1 < 3. When 
J2/J1 > 3, the ground state solution at J2/J1 = 3 is 
given for the entire range. 

Figure 3 shows the phase diagram obtained by solv- 
ing the equations (22). In all phases, Qc ■ z — 0. When 
J2 = 0, the triangular layers are decoupled and each 



A. Phase Diagrams for SGi 

1. 0-flux Ansatz 

The phase diagram for the 0-flux ansatz is given in Fig. 
5. In the quantum limit {k~^ ^ 1), we have three spin 
liquid phases: the 2D-Z2, 3D-Z2, and 3D-U{1) states. 
In the classical limit (k~^ <C 1), there are three long- 
range orders: the 120° Neel, coplanar spiral, and coUinear 
Neel states. The mean-field parameters of these states 
are depicted in Fig. 6 and 7 for particular values of k: 
= 9 for the spin liquid states and k^^ = 1 for the 
long-range ordered states. The six phases and the phase 
transitions among them are discussed below. 

a. 2D-Z2 Spin Liquid In the limit of small J2I Jij 
only the mean-field parameters in the triangular lattice 
plane, rjia and 771^3, are non-zero, while the inter-plane pa- 
rameters rj2a = ^72^ = 0. This yields a two-dimensional 
Z2 spin liquid. Furthermore, since ryi^ = 771/3, it retains 
the three-fold rotational symmetry, and is in fact one of 
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0.5 1 1.5 2 2.5 3 3.5 4 

FIG. 5. (Color online) Mean-field phase diagram of the 0- 
fiux ansatz. In the phase diagram, there are three long range 
orders for large k, which are the 120° Neel, coplanar spiral, 
and coUinear Neel states, and three spin liquid phases for 
small K, which are the 2D-Z2, 3D-Z2, and 3D-U{1) states. 
The thick blue line indicates the region where the 120° Neel 
state appears. This mean-field phase diagram recovers the 
classical phases in Fig. 3 in large k limit. At the boundary 
between the coplanar spiral and 2D-Z2 states (denoted with 
the dotted line), the phase transition is discontinuous, while 
a transition at any other phase boundary is continuous. 



the bosonic spin liquids on the triangular lattice discov- 
ered in earlier studies. '^^ The minimum single-spinon gap 
occurs at the corners of the Brillouin zone, at the points 
±K. Furthermore, the Z2 gauge flux excitations (visons) 
are also gapped. 

b. 3D-Z2 Spin Liquid (nematic spin liquid) Upon 
increasing J2/J1 further, the inter-plane couplings rj2a 
and 772/3 become non-zero, along with the intra-plane 
ones, rjia and rjip. Since these parameters all have 
different magnitudes in general, they break the 120° 
lattice rotation symmetry. We call such a spin liquid a 
nematic Z2 spin liquid. As the out-of-plane couplings 
are also non-zero, this is a gapped three-dimensional 
Z2 spin liquid. The minimum of the single-spinon gap 
shifts, from the corners of the Brillouin zone at the ±K 
points, to the center of an edge of the Brillouin zone at 
the M point, as J2/J1 increases. 

c. 3D-U{1) Spin Liquid When J2/J1 > 1.3 (for in- 
stance, at K~-^ = 9), a three-dimensional U{1) spin liquid 
is stabilized. In this phase, the parameters in the triangu- 
lar plane are zero, rjia = = 0, while the out-of-plane 
parameters, ri2a = V2p 7^ 0. We immediately notice that 
the links on which the parameters are non-zero have a 
bipartite structure. Hence, we can define staggered U{1) 




FIG. 6. (Color online) Mean-field parameters of the 0-flux 
spin liquid states at k.^^ — 9. In the 2D-Z2 spin liquid state 
(J2/J1 < 0.7), 771Q = 771/3 7^ and 772^ = Vifi = 0. The oppo- 
site is true for the 3D-U{1) spin liquid state (J2/J1 > 1.3). In 
the 3D-Z2 spin liquid state (0.7 < J2/J1 < 1-3), every mean- 
field parameter is nonzero. These features in the mean-field 
parameters indicate that the 3D-Z2 spin liquid state breaks 
the 120° lattice-rotation symmetry, while the rotation sym- 
metry is preserved in the other two states. 

transformations of the following form: 

bifj. e^'^bifi for i G A sublattice, 
bifi e""^6i^ for i G B sublattice, (23) 

for (j> G [0,27r). In other words, one can assign a posi- 
tive gauge charge to one sublattice, and a negative gauge 
charge to the other. This translates into the fact that 
the IGG is no longer Z2, but U{1), and we have a three- 
dimensional U{1) spin liquid. In three dimensions, such a 
phase may be stabilized, unlike the two-dimensional case. 
Analysis beyond mean-field shows that low-energy fluc- 
tuations of this U{1) spin liquid include two linearly dis- 
persing photons, and hence this phase is gapless.''^"'^^ The 
single-spinon gap reaches its minimum at the M point, 
the center of an edge of the Brillouin zone. 

d. Long-range orders Upon increasing k, the spinon 
gap collapses, and condensation occurs, leading to long- 
range magnetic order. Depending on the ratio of J2/J1, 
different kinds of magnetic orders are obtained. These 
are (1) 12(P non-collinear Neel order: at J2 = 0, we have 
decoupled triangular lattices. For finite but moderate k 
(> 1/3), this supports 120° magnetic order, which is con- 
tinuously connected to the classical limit. (2) coplanar 
spiral order: on increasing J2/J1, the ordering wavevec- 
tor for the spiral changes, and assumes an incommensu- 
rate value, except at special values of J2/Ji- However, 
the ordering is coplanar, and the ordering wavevector is 
in the x-y plane. While the classical solution (k = 00) 
is degenerate in this regime, at finite k, a particular or- 
dering wavevector Q is chosen through quantum order by 
disorder, as shown in Fig. 4 (blue line). (3) collinear Neel 
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FIG. 7. (Color online) Mean-field parameters of the long- 
range ordered states in 0-flux ansatz at = 1. In the 120° 
Neel state at J2 — 0, rjia = rj-^p ^ and ri2a = »72,3 ~ 0. The 
opposite is true for the collinear Neel state when J2/J1 > 2.7. 
The intermediate coplanar spiral state has a nonzero value for 
every mean-field parameter. 

order: At large J2 / Ji , the out-of-plane couplings dom- 
inate, and since these couplings have a bipartite struc- 
ture, they stabilize a two-sublattice collinear Neel order. 
Mean-field parameters for the above long-range ordered 
states at = 1 are plotted in Fig. 7. 

Phase Transitions. 

In Fig. 5, the transition between the incommensu- 
rate spiral and the 2D-Z2 spin liquid phases is first- 
order, while the rest are second order. These second 
order transitions were studied by Chubukov et al?^ in 
two dimension. Unlike the usual Landau theory of phase 
transition, here the coarse grained action is not written 
in terms of the magnetic order parameter, but in terms 
of the low-energy spinous b\^^ the U{1) gauge field, and 
the gauge-charge-2 Higgs scalar field 77.^. Now, we will 
briefiy describe these transitions. Different phases de- 
pend on which of the above bosons are condensed. 

1. Neel - 3D U{1) : The collinear Neel phase is 
described as a condensate of the spinous when 
the gauge-charge-2 Higgs field is gapped. The 
U{1) gauge field is rendered massive through 
the Anderson-Higgs mechanism. ^^'^^ The transition 
from such a phase to a gapped U{1) spin liquid is 
obtained by un-condensing the spinous while keep- 
ing the charge-2 Higgs field gapped. While this 
transition is continuous in the mean field level, it is 
known that fluctuation effects due to the gauge field 
may render this transition discontinuous in three 
dimensions.''" 

2. Neel - Spiral: In contrast, the transition from the 



FIG. 8. (Color online) Mean-field phase diagram of the n- 
fiux ansatz. At small J2/J1, the 120° Neel (at small k~^) and 
2D-Z2 spin liquid (at large k,~^) phases are found. Upon in- 
creasing J2/J1, there is a first-order transition (denoted with 
dotted line) into the collinear Neel (at small n"^) or 3D-U{1) 
spin liquid (at large states. The other transitions are 

second-order. In the phase diagram, every phase preserves 
the lattice-rotation symmetry, unlike the 0-fiux phase dia- 
gram in Fig. 5, where there are intermediate phases with 
broken lattice-rotation symmetry, such as coplanar spiral and 
3D-Z2 spin liquid states. 

collinear Neel to spiral phase is obtained by con- 
densing the charge-2 Higgs scalar in the background 
of the spinon condensate. 

3. U{1) - Z-2,: Starting from the C/(l) spin liquid, one 
can condense the chargc-2 Higgs scalar, and this 
breaks down the gauge group from U(\) to Z^. 
However, since the spinous are still gapped, this 
is a Z2 spin liquid phase. 

4. Z2 - Spiral: Finally, the transition from the Z2 spin 
liquid to the spiral phase is obtained by condensing 
the spinous in the background of a charge-2 con- 
densate, thereby completely gapping out the gauge 
fluctuations. 



2. n-flux Ansatz 

The phase diagram for the 7r-fiux phase is shown in 
Fig. 8. No new phases are present; however, the incom- 
mensurate spiral phase at large k and 3D-Z2 spin liquid 
at small k arc missing in the intermediate J2/J1 range. 

We note that none of the phases in the phase diagram 
(Fig. 8) break the lattice rotational symmetry. In addi- 
tion, the mean-field parameters in the triangular planes 
are never simultaneously nonzero along with the inter- 
plane parameters. At small J2/J1, the 120° Neel (at 
large k) and 2D-Z2 spin liquid (at small k) phases are 
found. Upon increasing J2/ Ji, there is a first-order tran- 
sition into the collinear Neel (at large k) or 3D-U{1) 
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FIG. 9. (Color online) Ground state energy per site (tgr) of 
the 0-flux and tt-Aux ansatze. egr is defined in (D18). Left: 
K,~^ = 9 (spin liquid regime). Right: = 1 (long-range 

order regime). The above plots of egr for both ansatze show 
that the 0-flux ansatz (blue) is generally more stable than the 
TT-flux ansatz (green). 



spin liquid (at small k) states. This is unlike the 0-fiux 
phase diagram, where there are intermediate incommen- 
surate spiral and 3D-Z2 spin liquid phases. Hence, in 
the K — 00 limit, the magnetically ordered phase is not 
the same for intermediate J2 / Ji as obtained from the 
classical analysis. 



B. Phase Diagram for SG2 

The phase diagram obtained for the SG2 ansatz is the 
same as for the tt-Aux ansatz for SGi, since none of 
the phases obtained break rotational symmetry. Con- 
sequently, both of the ansatze have identical mean-field 
solutions. 



VII. TWO-SPINON EXCITATION SPECTRA 

Having already discussed the ground state properties 
of the various spin liquid phases in the previous sec- 
tion, wc briefly discuss the two-spinon excitation spectra 
for various spin liquid states that we find for the 0-flux 
ansatz with SGi. Such a two-spinon excitation spectrum 
is gauge invariant (unlike the spingle spinon spectrum) 
and is proportional to the spin-structure factor measured 
in the neutron scattering experiments. ^^'''■^ Hence, it re- 
flects the physical symmetries broken in a phase. In our 
case, the two-spinon spectrum provides a valuable infor- 
mation for investigating the symmetries of a spin liquid 
state, especially the broken lattice-rotation symmetry in 
the nematic spin liquid state. 

The lower edge of the two-spinon spectrum is given 

by41.42 



0-flux ansatz vs. n-flux ansatz 

Before moving to next subsection, we compare the 0- 
flux and 7r-flux ansatze in terms of the ground state en- 
ergy and symmetry. Figure 9 shows their ground state 
energy per site as a function of J2 / Ji for two particular 
cases: = 9 and = 1. The ground state energy 
per site (e^r) is defined in (D18). From both cases in 
the figure, it is clearly seen that the 0-fiux ansatz (blue) 
is generally more stable than the 7r-fiux ansatz (green). 
In fact, there are two parameter regions where the two 
ansatze have the same energy: the small J2/J1 region 
and large J2/J1 region. In these regions, they have phys- 
ically identical mean-field solutions, which preserve the 
lattice-rotation symmetry, in the 2D-Z2, 3Z)-[/(l), 120° 
Neel, and collinear Neel states. However, when Ji and 
J2 are comparable, two ansatze have different mean-field 
solutions with different symmetries as discussed in Sec. 
VI A 1 and VI A 2. In the 0-flux ansatz, the 315-^2 and 
coplanar spiral states with broken rotation symmetry are 
stabilized. However, the 7r-flux ansatz allows only the 
other states with the rotation symmetry. Energetically, 
the broken symmetry states are more stable than the 
symmetry-preserving states, as shown in Fig. 9. 

From the above energy comparisons, we find that (i) 
the 0-flux ansatz provides more stable states compared 
to the TT-flux ansatz and (ii) the system is stabilized by 
breaking the lattice-rotation symmetry when the system 
is frustrated due to competing interactions Ji and J2. 



£;2(k) =min[e(p) + e(k-p)], (24) 
p 

where e(k) = min{aji(k), aj2(k)} and aji.2(k) are two en- 
ergy bands of the single-spinon excitations. For more de- 
tails about wi^2(k), the reader is referred to Appendix D. 
i?2(k) of the 0-flux state is plotted in Fig. 10 for several 
values of J2I J\ at = 9. The figure clearly shows that 
the 3D-Z2 state breaks the 120° lattice-rotation symme- 
try, (see (c),(d)) whereas the 2D-Z2 and 2>D-U{1) pre- 
serve the rotation symmetry (see (b),(e)). In addition, 
it is observed that i?2(k) continuously changes as J2/J1 
increases. The 2D-Z2 spin liquid state (J2/J1 < 0.7) has 
the minima of £^2(k) at the ±K points. In the 3Z)-Z2 
spin liquid states, (0.7 < J2/J1 < 1-3) the two minima 
in the two-spinon spectrum gradually move toward the 
r point as J2/^i increases. When J2/J1 = 1.3, the two 
minima merge at the F point, which is the minimum 
of the two-spinon spectrum of the ?>D-U{\) spin liquid 
(J2/J1 > 1.3). The continuous change in the minimum 
points is depicted in Fig. 10 (a). The minimum points 
of each spin liquid state are consistent with the ordering 
wave vectors of the corresponding long range order state. 

The existence of the 0-flux 3D-Z2 spin liquid state 
is an example of a nematic spin liquid that breaks 
the 120° rotational symmetry, but retains time-reversal, 
spin-rotational, and all other symmetries of the lattice. 
However, since the spinous may remain in a deconfined 
phase, such a state retains the fractionalized spin-1/2 
excitations and topological order that comprise the ex- 
otic behaviours of symmetric spin liquids. Its energetic 
favorability within mean-field theory, combined with its 
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FIG. 10. (Color online) The lower edge of the two-spinon excitations, i52(k), of the 0-flux spin liquid states at = 9. i52(k) 
is defined in (24). We plot the plane kz — 0, which contains the minima of these excitations, (a) The positions of the minima 
in i?2(k) in the first Brillouin zone as a function of J2/J1. (b) The 2D-Z2 spin liquid, (c) ~ (d) The 3D-Z2 spin liquids, (e) 
The 3D-U{1) spin liquid. The two minimum points at ±K gradually move toward the F point and then merge at the point as 
J2/J1 increases. In each plot, the hexagon denotes the first Brillouin zone at k, = 0. 



necessity to describe the spiral order, indicates that such 
a state may be relevant for a model with the 6H-B struc- 
ture. Although the Schwinger boson mean-field theory 
predicts that spiral order is stabilized over a large range 
of K, such a nematic spin liquid serves as a starting point 
for understanding spin-disordered ground states of the 
frustrated 6H-B structure. 



VIII. SUMMARY AND OUTLOOK 

In this work, we have analyzed the possible three di- 
mensional bosonic spin liquids on an alternately stacked 
triangular lattice structure. For the frustrated Heisen- 
berg Hamiltonian with nearest neighbour intra- and 
inter-planc couplings, wc find several three dimensional 
quantum spin- liquid phases for small values of k (~ 25). 
Our PSG analysis significantly constrains the number of 
possible Z2 spin liquids on this 6H-B lattice. It gives 
two kinds of gauge-inequivalent anzatz (0-flux and 7r-flux) 
that are consistent with different lattice symmetries. A 
central finding of our analysis is a 3D nematic Z2 spin liq- 
uid that breaks lattice rotation symmetry that is reflected 
in the spin-structure factor. Interestingly, our Schwinger 
boson mean field theory calculation shows that this spin 
liquid is not only energetically more stable (within self- 
consistent mean-field theory) than an isotropic one in the 
same parameter regime, but is also naturally connected 
to the classical limit of coplanar spiral order. We note 
that spiral order also breaks lattice rotation symmetries 
along with spin rotation. It is interesting that this spon- 
taneously broken lattice rotation persists in the nematic 
spin liquid, even though the spin-rotation invariance is 
restored across the transition. Though our PSG analysis 
is aimed to capture Z2 spin liquids, we identify a bipar- 
tite structure of the mean-field parameters in one of the 
spin liquid phases. The resultant phase actually turns 
out to be a gapped U(l) spin liquid in three dimensions. 

In the present self-consistent mean-field calculations of 
the given spin-model, the maximum k ~ 25* for which the 
various spin liquids are stable is rather small compared to 
the physical value of the spin. However, the spin liquids 
found here may be stabilized by further neighbour inter- 



actions or multi-spin exchange processes for higher values 
of K. It would be interesting to see the fate of the nematic 
spin liquid in such models. Another interesting feature 
of the phase diagram is the existence of various uncon- 
ventional quantum phase transitions between the mag- 
netically ordered and disordered phases. Most of these 
transitions turn out to be continuous within our mean- 
field theory. We identify these transitions by extending 
the existing framework^"'' in terms of condensation of 
spinous to three spatial dimensions. 

It would be interesting to explore, in addition to 
Ba3NiSb209, other possible Mott-insulators on three di- 
mensional AB-stackcd triangular lattices that may sta- 
bilize one or more of the spin liquid phases described 
here. Of particular interest would be the materials close 
to metal-insulator transition where charge fluctuations 
may enhance further neighbour interactions and multi- 
spin processes. 
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Appendix A: Projective Symmetry Group 
Construction I 

We split the determination of the PSG into two parts; 
the first described the procedure and results, leaving 
most of the details to the second. 

In this Appendix, we introduce twenty five multiplica- 
tion rules among the seven symmetry operations given in 
(3), construct the PSG corresponding to the three trans- 
lations, and present the final PSG for both symmetry 
groups SGi and SG2. Wc leave the derivation of the 
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PSG elements for the other symmetry group operations 
to Appendix B. 

The multiphcation rules among the seven symmetry 
operations in (3) are given by the following: 



TaTs = T3T2, 
T1T3 = T3Ti, 

TiHi =niTf\ 

TiHa = n2ri, 



TiE = ET-\ 



T2R = RTi, 

T3R — RT3, 

R^ ^ I, 

Rill = TliR^\ 

RII2 TI2R, 

RE = ET,-^T^^R, 



(Ala) 
(Alb) 
(Ale) 

(A2a) 
(A2b) 
(A2c) 
(A2d) 

(A3a) 
(A3b) 
(A3c) 
(A3d) 
(A3e) 

(A4a) 
(A4b) 
(A4c) 
(A4d) 
(A4e) 
(A4f) 

(A5a) 
(A5b) 
(A5c) 
(A5d) 
(A5e) 
(A5f) 
(A5g) 



where / is the identity operator. The above multiplica- 
tion rules provide constraints on the PSG of the symme- 
try operations. 

We now construct the PSG for the subgroup of trans- 
lations defined by the equation (Al), re-writing it as 

Tf = /, (A6a) 

= /, (A6b) 
Tf ^TaTiTg-i = /. (A6c) 

By considering the equivalent expression for PSG ele- 
ments, as in (14), from (A6a) we obtain 

( Gti Ti ) - 1 ( Gt, T2 ) {Gt, Ti){Gt,T2)-^ 
= [T^'G^lTi) ■ (T-'Gt.Ti) 

■mT^')-'GTATiT2'))-G^l 
e IGG-:Z2, 



where Gx {X = Ti,T2) is the gauge transformation as- 
sociated with the operation X in the SG. Under Gx , the 
boson operator acquires the phase e**^ '""^ . The phase of 
the above equation can be written as 

-(/.Tjri(r)]+</.T,[ri(r)] 

+(/.Ti [TiT^\v)] - <i>T, [r] = 71171, (A7) 

where ni G {0, 1} and ??i — (n\ ~ 1) corresponds to 
the element +1 (—1) in the Z2 gauge group. Equation 
(A7) is the first constraint on the PSG. Whenever we 
derive a constraint on the PSG from a multiplication rule 
among the symmetry operations, we employ an integer 
variable, like n\ in the above equation. Hereafter, (i = 
1, • • • ,25) is regarded as an integer variable, which is 
either or 1. It must be noted that in all phase equations 
like (A7), equality is to be understood as modulo 27r, due 
to the 2ti periodicity in the phases. 

Determining the possible solutions of the constraint 
equations like (A7) can be facilitated by an appropriate 
choice of gauge fixing. For the first gauge fixing, we set 



'/>Ti(7-l,?'2,?'3)p = 0, 



(A8) 



for both sublatticcs {p = A,B). Then, (A7) is reduced 
to 

<l>T2iri,r2,r3)p = (/its (0, ?-2, ra)^ + niTrri. (A9) 
Taking another gauge fixing 

</'T.(0,r2,r3)p = 0, (AlO) 

we obtain 

<t)T2{'ri,r2,r3)p = ninri. (All) 

In a similar way, (r) is determined from (A6b) and 
(A6c). If we use (A8) and (All) and take the gauge- 
fixing 

'/'T3(0,0,r3)p = 0, (A12) 

then we are lead to 

(f>T3{ri,r2,r3)p = n27rr2 +n37rri. (A13) 

We must ensure to fix the gauge freedom so as to sat- 
isfy (A8), (AlO), (A12) at the same time. The gauge- 
fixings can be realized in the following way.''' Suppose 
that (f)ip^ {n — 1,2,3) is the initial choice for the phase of 
Gt„ ■ Then, we consider a series of gauge transformations 
Gm {m = 1, 2, 3), so that 



where Gm = e*'^'^™ (m 
(l)Gm{'ri,r2,r3)p 



(Al4a) 

: 1, 2,3) is defined as follows: 

n 

X] '^TT"^^(*'''2,f3)p 

t — — 00 

j=-oo 

f2 4r"(0'0'fc)f (A14b) 
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It is important to note that 4>Gm depends on c/)^" ^\ 

To determine 0^"^ (m = 1,2,3), it is necessary to un- 
derstand how an arbitrary gauge transformation changes 
Gx- It can be understood from the fact that our mean- 
field ansatz is invariant under the PSG, {GxX}. Then, 
if we take a gauge transformation G, the transformed 
mean-field ansatz is invariant under G ■ GxX ■ G~^ = 
Gx ■ X, where Gx =G Gx ■ XG-^X-\ This gives us 
a new PSG, {Gx ■ X}. Then, under the transformation 
G, 4>x [r] changes in following way. 



0G[r]+0A-[r]-0G[^-'(r)]. 



(A15) 



According to above transformation rule, (px [r] acquires 
the additional phase '?I>G[r] ~ (t>G[X~^{v)] under the trans- 
formation G. Applying the above rule to each transfor- 
mation in (A14), we find that the gauge fixings (A8), 
(AlO), (A12) are all satisfied. As seen in the definition of 
(A14), the above gauge-fixing is defined for open bound- 
ary conditions. Periodic boundary conditions introduce 
additional subtleties, so we consider the PSG for systems 
with open boundary conditions, with the understanding 
that the particular boundary condition will be irrelevant 
in the thermodynamic limit. 

Readers who are interested in detailed derivation of 
Gx for the rest of the symmetry group are referred to 
Appendix B. We present the phases 4>x of Gx for the 
full PSG below. For SGi, 



'/'Ti(ri, 




, = or TT, 


(A16a) 


'T2,3(''l, 


r2,r3)i 


, =0, 


(A16b) 








(Al6c) 


'/'n2(?'i, 


r2,r3), 


, = (m2 + TO3 • <5p,b) tt. 


(A16d) 




r2,r3)i 


TT 

2' 


(Al6e) 



Gonsidering the remaining rotation R, we obtain the PSG 
for the SG2, which consists of (AI6) and (A17). 



1. (t>ni{ri,r2,r3)p 

Rewriting (A2) into constraints like (A7), and solving 
the resultant equations, we obtain 

0ni(?'i,r2,r3)p 

(j)ni (0, 0, 0)p + 7i47rri + ?i57rr2 + neTrra 
ni 



and 



+ Y'^^^'^ -^^2)- "37rr'2r3, 



20ni(O,O,O)p = n77r, 
ni = 714 = 0. 



(Bl) 



(B2a) 
(B2b) 



Of the above equations, (Bl) is determined by (A2a,b,c) 
and (B2) by (A2d). As mentioned earlier, integer 
variables (n^) are introduced when multiplication rules 
among the symmetry operations are used to place a con- 
straint on the PSG. In the above expressions, the integer 
variable can be eliminated by the gauge transforma- 
tion G4 = e^'^'^i , where 

(f>Giiri,r2,r3)p = irri. 

Under this transformation, 

0T„(?'i,r2,r3)p (j)T„{ri,r2,r3)p + (5„,i7r {n = 1,2,3), 
0ni(?'i,?'2,r3)p -> 4>Ui{ri,r2,r3)p - 7rr2. 

If 77,5 = 1, n^T:r2 in (Bl) is removed under the transfor- 
mation. Combining the above results with (A8), (All), 
(A13), the PSG for {Ti, T2, T3, Hi} is described as fol- 
lows: 

0Ti(ri,r2,r3)p = or tt, 

(t)T2{ri,r2,r3)p 0, 

(l)T3i'ri,r2,r3)p = n27rr2 -l- nsirri, 

0ni(ri,r2,r3)p = (0, 0, 0)p 4- neTrrs -n3nr2r3, 



where 



(f>R{ri,r2,r3)p = 171477. 



(A17) 



20ni(O,O,O)p = n7^. 



In (AI6) and (A17), rrii (i = 1,2,3,4) are new integer 
variables employed to simplify the final expression of the 
PSG and m, e {0,1}. 



Appendix B: Projective Symmetry Group 
Construction II 



In this section, we derive the transformation Gx ~ 
gj0x fQj. |;]-^g remaining elements of SGi and SG2 beyond 
the translations. 



2. <Pn2{ri,r2,r3)p 

Gonducting similar calculations as above for (A3) leads 
to the following equations: 

<Pn2{ri,r2,r3)p = 0n2(O,O,O)p 

+ TigTrri + ?i97rr2 + nioTrrs, (B3) 

2(j)n2 (0, 0, 0)p = (nio • (5p,B -I- nii)n, (B4) 

n3 = m = m = ni2 = 0. (B5) 

(B3) is determined by (A3a,b,c), (B4) by (A3d), and 
(B5) by (A3e). Using this result, the PSG for 



13 



{ri,r2,T3,ni,n2} is given by 

(f>Tiiri,r2,r3)p = or tt, 

<l>T2{ri,r2,r3)p = 0, 

(f>TAri^f2,r3)p = n27rr2, 

</>ni {ri,r2,r3)p = (0, 0, 0)p, 

0n2(^i:^2,?'3)p = 0n2(O, 0,0)p,+ng7rr2 +?T.io7rr3, 

where 

20ni(O,O,O)p = n77r, 

20n2 (0, 0, 0)p = (nio • (5p,B + »^ii)7r. 



Applying the equation (13) to the above nonzero mean- 
field parameters, we can find additional constraints on 
the six integer variables. First, wc note that the link 
(0, 0, 0)a — >■ (0, 0, 0) B does not move under the operation 
Hi. Using the equation (13) with X = Hi, i = (0,0, 0)a, 
j = (0, 0, 0)b, we have 

</.ni(0,0,0)A + (/)ni(0,0,0)s -0. (BIO) 

Next, we note that for both of the sublattice p = 
the links, (0,0, 0)p (-1,0, 0)p and (0,0, 0)p -> 
(1, 0, 0)p are connected (i) by IIi or (ii) by the interchange 
of the two sites and Ti. In this case, the equation (13) is 
written as follows: 



3. (t)s{ri,r2,r3)p 

Repeating the same procedure for (A4), we obtain 

<l>s{ri,r2,r3)p = (/)h(0,0,0)p 

+ nisTrri + ni47rr2 + ni57rr3, (B6) 

0h(O, 0, 0)p + 0h(O, 0, 0)p- = ni67r, (B7) 

ni3 = nn = 0, (B8) 

and 

n2 = nio = ni5 0, (B9a) 

(0, 0, 0)s = (0, 0, 0)a + nisn. (B9b) 

The first equation (B6) is determined by (A4a,b,c), (B7) 
by (A4d), (B8) by (A4e), and (B9) by (A4f). Now we 
have the PSG for {Ti, T2, Tg, Hi, 02, S}, which is de- 
scribed by 

4>Tiiri,r2,r3)p = or tt, 
<l>T2{ri,r2,r3)p = 0, 
(f>T3{'^li'^2,r3)p = 0, 

(t)nAri,r2,r3)p = ?ini (0, 0, 0)p, 
(t>n2{ri,r2,r3)p = (^nj (0, 0, 0)p, +n97rr2, 
(I}s{.ri,r2,r3)p = 0h(O, 0, 0)p, -^ni47rr2. 



. g-i0ni[ni(O,O,O)p]-i</.ni[ni(-l,O,O)p] 



where 



20ni(O,O,O)p = 717^, 
2(^n2(0,0, 0)p = niiTT, 
0h(O,O,O)a + 0h(O,O,O)b = ni67r, 
(0, 0, 0)b = (0, 0, 0)a + "is^- 

There arc six integer variables in the PSG obtained 
above. However, in determining possible mean-field 
Hamiltonians, some of those variables can be eliminated 
by using symmetries. To be specific, we assume 

^(0,0,0)a->(1,0,0).4 7^ 0' 
'?(0,0,0)a->(14,0).4 7^ 0' 

'7(o,o,o)A->(o,o,o)i3 7^ 0, 

V{0,0,0)a->{1A,0)b ^ 0- 



'7(0,0, o)p^-(i,o,o)p 

= ''?(0,0,0)p-^(-l,0,0)p 

= ^'7(o,o,o)p-i-(-i,o,o)p 

^ 4>n, [ni(0, 0, 0)p] + (/)ni [ni(-l, 0, 0)p] = TT. 

Arranging the above equation, wc get the condition 

20ni(O,O,O)p = 7r (p = A,S) 
=> nj = 1. (Bll) 

Then, i^ni is determined by (BIO) and (Bll): 

0ni {ri,r2,r3)p = (0, 0, 0)p 

= Q+mi+VB^7r, (B12) 

where mi G {0, 1}. 

Next, we notice that the links, (0,0, 0)a (1,0, 0)^ 
and (0,0, 0)a — >■ (1,1, 0)a are invariant under 112. From 
this fact, 

0n2(O,O,O)A + 0n2(l,O,O)A = O, 

nil = 0, (B13) 

and 

(/)n2(0,0,0)A + (/>n2 (1,1,0)^ = 0. 
=> ng = 0. (B14) 

With these conditions, (f)ii2 can be written as 

(l}U2{ri,r2,r3)p = (?!)n2 (0, 0, 0)p 

= (m2 -f ma • (5p,B)7r, (B15) 

where m2, ma G {0, 1}. 

Finally, let us consider the link (0,0, 0)s — > (0, 0,0)a- 
This link can be obtained by acting S on the link 
(0,0, 0)a (0,0, 0)b. In addition, (0,0, 0)^ ^ 

(0,0, 0)a = -(0,0, 0)a ^ (0,0, 0)b. From these, wc ob- 
tain 

</'h(O,O,O)a + 0h(O,O,O)b =^ 
=^ nie = 1. (B16) 

We also note that the links (0,0, 0)a —5- (1,1, 0)b and 
(-1,-1, 0)a ^ (0,0, 0)b are connected by T^^T^^, 
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and the links (0,0, 0)^ (1,1, 0)s and (0,0, 0)b 
(— 1,— 1,0)a are connected by S. It follows that 

0H(O,O,O)B + (/)H(-l,-l,O)A = 7r 

ni4 = 0. (B17) 

From the conditions (B16) and (B17), (j>s can be written 
as follows: 

"00 {p = A) 

TT - 00 {P = B) 



(B18) 



The dependence of 0h on the sublattice can be eliminated 
by the gauge transformation G5 = e^'^^s ^ where 



01 iP^A) 
-01 iP = B) 



Under the transformation, only 0e: changes, in the fol- 
lowing way: 

0H(ri,r2,r3)p -> 0H(ri,r2,r3)p + 20i • (-l)'^''-«. 
By choosing 2(f>i — tt /2 — (f>Q, we obtain 



fe(''l,''2,''3)p = 



(B19) 



Collecting these results, we obtain (AI6), giving the PSG 
for the SGi. 



Appendix C: Brief sketch of the derivation of the 
0-flux and 7r-flux ansatze 



In this section, we construct the ansatze of the SGi 
by using (13) and (A16). This generates the values of 
all symmetry-related mean-field parameters from a single 
bond, through the application of elements of the PSG. 
For example, if we set X = Tn {n = 1, 2, 3) in (13) and 
use (Al6a) and (Al6b), then we have 



^r„(j)T„(i) = Vij- 



(CI) 



This means that the mean-field ansatze are translation- 
ally invariant. Therefore, it is enough to specify the 
mean-field structure within a unit cell. 

Considering the unit cell denoted with the ellipse in 
Fig. 1, the twelve links consist of six intra-laycr links 
(C2) and six inter-layer links (C3). 



(0,0, 0)a 
(0,0, 0)a 
(0,0, 0)a 
(0,0, 0)b 
(0,0, 0)b 



(1,0,0)a, 
(0,1,0)^, 
(1,1,0)^, 
(-1,0,0)b, 
(0,-1, 0)b, 



(0,0,0)b^(-1,-1,0)j 



(C2a) 
(C2b) 
(C2c) 
(C2d) 
(C2e) 
(C2f) 



4. (f)R{ri,r2,r3)p 

We now consider adding the rotation R to the symme- 
try group, in order to find the PSGs for SG2. We obtain 
following conditions from (A5a,b,c). 

= 0_r(O, 0, 0)p -t- nigTrri + n2o'^r2 + n2iTrrj,. (B20) 

The constraint from (A5d) reduces the above expression 
to 



4'Riri,r2,r3)p = 0^(0, 0, 0)p + n2o7rr2. 



(B21) 



where 



30ij(O,O,O)p = n227r. (B22) 

n22 G {0, 1} is introduced when (A5d) is written in terms 
of 0i?(ri, r2, 7^3 )p. By using the constraint from (A5e), we 
can remove the r2-dependence in cjjfj: 

Mn,r2,r3)p = M0,0,0)p, (B23) 

with (B22) and 

2(j)R{0,0,0)p = -n23TT. (B24) 

n23 & {0, 1} is introduced when (A5e) is used. (A5f,g) do 
not give a new condition on the integer variables. There- 
fore, we have following expression for 0^: 



4>R{ri,r2, r-i)p = (7122 + n2'i)Tr. 



(B25) 



This can be simplified to (A17), which is the last piece 
describing the PSG for the SG2. 



(0,0, 0)a 
(0,0, 0)a 
(0,0, 0)a 
(0,0, 0)a 
(0,0, 0)a 
(0,0, 0)a 



(0,0, 0)b, 
(1,1, 0)b, 
(0,1, 0)b, 
(0,0, -1)b, 

(1,1,-1)5, 
(0,1, -l)s. 



(C3a) 
(C3b) 
(C3c) 
(C3d) 
(C3e) 
(C3f) 



The mean-field parameters are assumed to be real-valued 
to preserve the time reversal symmetry. Since "qij ~ ^Vjij 
there is the positive direction a nonzero the mean-field 
parameter at each link Thus, a mean-field ansatz 

is depicted with arrows indicating the positive direction 
at each link. 

For the ansatze having the symmetries of SGi, we fix 
mean-field parameters at four links: (C2a), (C2b), (C3a), 
(C3b). The links (C2) and (C3) within the unit cell are 
connected by symmetry operations in following way. 



(C2a) 



(C2b) 



(C2d) 



(C2e) 



(C3a) 



(C3b) 



Tl2i 

(C3d) 



n2i 

(C3e) 



(C2c) 

h (C2f) 
i> (C3c) 

Tl2i 

i> (C3f) 



(C4) 



In the above diagram, the boxes denote the links where 
the mean-field parameter is fixed. If we apply (13) to 
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(C4) , we can determine the other eight mean- field param- 
eters in the unit cell. In this way, two distinct ansatzc 
are found: the 0-flux ansatz and n-flux. Their mean-field 
configurations in the unit cell are given as follows. 
0-flux ansatz 

ma = ??(0,0,0)a^{1,0,0)a = '7(-l,0,0)i5->(0,0,0)i5 (C5a) 
ViP = '7(0,0,0).4->(0,l,0).4 = V{1,1,0}a~*(0,0,0}a 

= '7(0,-1, o)b-*(o,o,o)b = '7(o,o,o)B->(-i,-i,o)£l!C5b) 

r]2a = ?7(0,0,0)A->(0,0,0)i3 = '?(0,0,0)a^(0,0,-1)b (C5c) 
1120 = ?7(1,1,0)b^(0,0,0)a = '7(1,1,-1)b^(0,0,0)a 

= ?7(o,i,o)b^(o,o,o)a '7(o,i,-i)b^(o,o,o)a (C5d) 
TT-flux ansatz 

Via = '7{0,0,0)a->(1,0,0)a = '?(-1,0,0)b^(0,0,0)b (C6a) 
Vlf! = ?7(0,0,0)a^(0,1,0)a = '?(1,1,0)a^(0,0,0)a 

= '7(o,-i,o)b^(o,o,o)b = '7(o,o,o)B^(-i,-i,o)iC6b) 

V2a = ??(0,0,0)a^(0,0,0)b = ^(0,0,0)a^(0,0,-1)b (C6c) 
?72/3 = '7(0,0, 0)a^(1,1,0)b = '7(0,0, 0)a^(1,1,-1)b 

= '7(0,0, o)a-^(o,i,o)b = '?(o,o,o)a->(o,i,-i)b (C6d) 

In the above expressions, the positive, real 
{'7iq, '71/3, '?2q5 ?72^} are fixed mean-field parameters 
at the links (C2a), (C2b), (C3a), (C3b), and the sub- 
scripts in the above expressions indicate the directions 
of positive mean-field parameter. These ansatze are 
depicted in Fig. 2. 

Appendix D: Mean-field Hamiltonians 

In this appendix, we provide the mean-field Hamiltoni- 
ans of the 0-fiux and 7r-fiux ansatze in momentum space. 
For each ansatz, the mean-field Hamiltonian takes the 
following form: 

Hmf = 2^„eeo + *^(k)D(k)*(k), (Dl) 



where 



eo = -A(K + l) 



— K 
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(D2) 



vE'(k) = (5^t(k),&Bt(k),6t (_k),6t (-k))^, (D3) 



/ 



D(k) = 



A 




C(k) a*(k) 
Vp*(k) C(k) 



C(k) p(k) 
a(k) C(k) 



A 





A 



(D4) 



In the above expressions, N^c is the number of unit cells 
and &m^(k) (to = A,B;fj, =t,i) is the boson operator 
in momentum space. Due to the fact that the 0-flux and 
TT-flux ansatzc arc translationally invariant, the Lagrange 



multiplier is taken to be uniform in the mean-field Hamil- 
tonian: Xi = A. Matrix elements of D(k) are defined as 
follows: 



ikt 



-ikt 



C(k) = -^Ji?7iQ 

tGTo 

p{k) = -\j2ma E ^'''■^ 



) 



(D5) 



(D6) 



— ik-v 



The sets in the above expressions are 

T„ = {Ri}, (D8) 
T^ = {R2,-Ri-R2}, (D9) 
14 = {0,-R3}, (DIO) 
= {Ri + R2, R2, Ri + R2 - R3, R2 - RsK-Dll) 

In the expression of (T(k), $ = for the 0-fiux ansatz and 
$ = TT for the TT-flux ansatz. 

The mean-field Hamiltonian for the symmetric ansatz 
is obtained by setting rjia = rjip = rji and ri2a = V2/3 = '72 
in the TT-flux mean-field Hamiltonian. 

The mean-field Hamiltonian (Dl) is diagonalized via 
the Bogoliubov transformation:"** 



«'(k) = M(k)r(k), 



(D12) 



where 



r(k) = (7it(k),72t(k),7iV-k),72\(-k))^. (D13) 

7„a(k) satisfies the bosonic statistics, which imposes a 
condition on the transformation matrix M(k): 



M(k)lBMt(k) =Is, 



(D14) 



where I^ = diag(l, 1, — 1, — 1). Due to this constraint, 
iBD(k) is diagonalized instead of D(k), so the eigenvalue 
problem has the following form: 

M"i(k)lBD(k)M(k) = iBrj(k), (D15) 

where r2(k) is the diagonal matrix of eigenvalues, 

fl{k) = diag(a;i^(k),a;2t(k), wi^(-k), W2i(-k)), 

(D16) 

In fact, w„-|-(k) = a;„4,(— k) = w„(k) {n = 1, 2), due to the 
time reversal symmetry of the mean-field Hamiltonian. 
In addition, a;„(k) = w„(— k) since D(— k) — D*(k). 
The eigenvectors are contained in the columns of ]V[(k) 
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and normalized according to (D15). By the above trans- 
formation, 



HmF = 2Nuc£gr + X! X! ^"(l^)7riQ(k)7na(k), 

(D17) 



k n=l,2 



where egr is the ground state energy per site; 



k ri=l,2 



The single-spinon excitation spectrum is given by aj„(k). 
If the spinon excitation becomes gapless at k = ±k*, 



then the spinon condensate is considered: 
x(k) = {^{k)) 



(k), a;st(k), a;^_^(-k), x;^^(-k)) ^ 



(D19) 



The condensate vector is a zero-energy eigenvector found 
from 



D(±k*)x(±k*) = 0, 
where x(±k*) is normaUzed to satisfy 



(D20a) 



K= ^ E 

k=±k* p=A,B 

+ E E (^j,p(k)&PM(k)>- (D20b) 

k/±k* p=A,B 

Then, the ground state is determined by solving 



dt 



gr 



dr] 



iv^Via,--- ,V2p), (D20c) 



together with (D20a) and (D20b). (D20) is the momen- 
tum space version of the self-consistent mean-field equa- 
tions (8). 
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